Zurl is an HTTP client daemon based on libcurl that makes outbound HTTP requests asynchronously. It’s super useful for invoking Webhooks. Zurl supports fire-and-forget invocation, error monitoring, and protection from evil callback URLs. Sounds pretty great, right? Let’s see how it’s done.

Installing

If you’re using Debian or Ubuntu, installing Zurl is easy:

sudo apt-get install zurl

Once installed, Zurl should be running in the background and listening on some FIFOs in /var/run/zurl.

By default, only users in the zurl group can access the server, so either put yourself in that group or change the ipc_file_mode property in /etc/zurl.conf from 660 to 666 and restart the server. Note: if you install from source then you’ll need to launch the server yourself, and the default configuration is a bit different.

Making a call

To make an HTTP request, send a simple JSON-formatted message to Zurl’s zurl-in endpoint containing the HTTP method and URL to hit. Here’s some example code in Python:

# send.py
import uuid
import json
import zmq

ctx = zmq.Context()
sock = ctx.socket(zmq.PUSH)
sock.connect('ipc:///var/run/zurl/zurl-in')

msg = {'from': 'testworker', 'id': str(uuid.uuid4())}
msg['method'] = 'POST'
msg['uri'] = 'http://example.com/path/'
sock.send('J' + json.dumps(msg))

That’s it! The above program will finish very quickly since all it does is send a message to Zurl and then exit. Zurl will perform the HTTP request asynchronously in the background. You can quickly confirm that the code works by using something like WebhookInbox for receipt.

Calling Webhooks via Zurl is great for many reasons:

  • The calling thread or process can exit without causing the HTTP request to be cancelled. Great for simple, fire-and-forget triggering.
  • The caller doesn’t block. This is useful in non-event-driven environments, where an HTTP request would normally block the calling thread.
  • Zurl protects you from calling evil URLs. You can use the policy configuration parameters in zurl.conf to tweak this.

Handling errors

“But wait!” you say, “All this fancy async stuff means we don’t know when a call fails”.

Don’t worry, we’ve got you covered! The zurl-out endpoint can be used to monitor for responses. What’s really cool is that this can be done by a completely separate process than the one that initiated the request.

Here’s some Python code for tracking HTTP responses:

# watch.py
import json
import zmq

ctx = zmq.Context()
sock = ctx.socket(zmq.SUB)
sock.connect('ipc:///var/run/zurl/zurl-out')
sock.setsockopt(zmq.SUBSCRIBE, '')

while True:
    m_raw = sock.recv()
    at = m_raw.find(' ')
    m = json.loads(m_raw[at + 2:])
    print json.dumps(m, indent=2)

All the above code does is pretty-print messages from Zurl. Here’s what the output looks like when receiving a response from WebhookInbox:

$ python watch.py
{
  "body": "Ok\n", 
  "code": 200, 
  "from": "{377bfeb9-81d1-48e7-af42-11c4dd3c6fd3}", 
  "seq": 0, 
  "headers": [
    [
      "Date", 
      "Fri, 03 Oct 2014 07:37:37 GMT"
    ], 
    [
      "Server", 
      "Apache/2.2.22 (Ubuntu)"
    ], 
    [
      "Vary", 
      "Accept-Encoding"
    ], 
    [
      "Content-Type", 
      "text/html; charset=utf-8"
    ], 
    [
      "Access-Control-Allow-Methods", 
      "OPTIONS, HEAD, GET, POST, PUT, DELETE"
    ], 
    [
      "Access-Control-Expose-Headers", 
      "Date, Server, Vary, Transfer-Encoding"
    ], 
    [
      "Access-Control-Allow-Credentials", 
      "true"
    ], 
    [
      "Access-Control-Allow-Origin", 
      "*"
    ], 
    [
      "Transfer-Encoding", 
      "chunked"
    ], 
    [
      "Connection", 
      "Transfer-Encoding"
    ]
  ], 
  "reason": "OK", 
  "id": "4e47c9b5-5d43-479c-a098-81f4a5471cf2"
}

If you wanted to look for errors, you could check for m['type'] == 'error' (transport level errors) and/or m['code'] != 200 (HTTP-level errors).

Most of the time, errors from Webhooks are used to delete broken subscriptions. This kind of thing can easily be done from a separate monitoring process such as the above. There’s no need to burden the code that is firing off the Webhook calls. Leave that code speeding along unhindered. Also, if your monitoring process fails for some reason (maybe your subscription DB is down), your Webhook calls are not impacted.

Want more context in a response message than just the request id, to make it easier to match up the response with an associated subscription? Set a user-data field in the request message containing any data you want:

...
msg['method'] = 'POST'
msg['uri'] = 'http://example.com/path/'
msg['user-data'] = {'subscription-id': '15c9b773'}
...

Zurl will echo this back in the response message:

  ...
  "code": 200, 
  "from": "{377bfeb9-81d1-48e7-af42-11c4dd3c6fd3}", 
  "seq": 0, 
  "user-data": {
    "subscription-id": "15c9b773"
  },
  ...

Happy Webhooking!